COMP30760 Data Science in Python Assignment -2¶
SURABHI AGARWAL - 17203535
- As a part of this assignment, I am going to scrape data from the following HTML page http://mlg.ucd.ie/modules/COMP30760/timeseries.html and analyse 4 stocks from the page given.
The Stocks Selected:
- Apple (APPL)
- Alphabet Inc. (GOOG)
- Disney (DIS)
- Microsoft (MSFT)
In [1]:
import pandas as pd
from pandas import Series
import urllib.request
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
import plotly.graph_objects as go
Apple
- Stock analysis for the company apple
Task 1: Data Collection¶
- Using python, I downloaded the HTML pages for my chosen stocks
In [2]:
#Defining a method called get_raw_data which requests data from the given html link for the company passed as parameter
def get_raw_data(company):
url = "http://mlg.ucd.ie/modules/COMP30760/stocks/"+company+".html"
try:
response = urllib.request.urlopen(url)
text = response.read().decode()
except:
print("Failed to retrieve %s" % url)
Task 2 : Data Preparation¶
Parsing data:
- Using pandas libray, I parsed and extracted the data for the company APPLE and represented the whole dataset as a dataframe
- Handling missing values
- Saving and exporting the data into csv format
In [3]:
#calling the method get_raw_data to get data for the company APPLE
get_raw_data("aapl")
#Creating Apple dataframe and concatenting all the columns into one dataframe
apple_df = pd.read_html("http://mlg.ucd.ie/modules/COMP30760/stocks/aapl.html")
apple_df = pd.concat(apple_df)
#Displaying the dataframe
apple_df
Out[3]:
Representing data as time series:
- To represent data as time series, first I would combine the year, month and day column of the data frame
- Use the date column as the index of the data frame
- Display the dataframe
In [4]:
#representing data as time series
apple_df['Date'] = pd.to_datetime(apple_df[['Year', 'Month', 'Day']])
#setting the data column as index
apple_df.set_index('Date', inplace =True)
apple_df.head()
Out[4]:
Exporting data to an appropriate format:
- Exporting the raw data into csv file with date column
In [5]:
#Exporting the data to csv file
apple_df.to_csv("Apple Stock.csv", index=True, encoding='utf8')
In [6]:
#Dropping the the unnecessary columns
apple_df = apple_df.drop(["Stock", "Year", "Month", "Day"], axis=1)
#Diplaying the data frame
apple_df.head()
Out[6]:
Handling missing values:
- Using interpolate method in the pandas library, I filled in the missing values in the dataframe
In [7]:
#Using df.interpolate, we can fill in the missing values in the dataframe
apple_df = apple_df.interpolate()
#Check if there are still any missing values left
apple_df.isnull().sum()
Out[7]:
In [8]:
#Rounding off the stock figures to make sure there is consistency in the data
apple_df = apple_df.round(2)
apple_df.head()
Out[8]:
Pre processing completed:
- The data contains no null values i.e there are no values in the dataset such as "Not Available", "NaN or "NA"
- All the values are not strings either
- This indicates that there all the values are present in the dataset.
Task 3.1: Data Analysis: Summarising the cleaned dataset¶
Descriptive Statistics:
- Displaying the descriptive statistics of the dataset
- Using the cleaned dataset for further analysis and visualization
- Grouping the data by year and taking mean of the values and displaying descriptive statistics
Calculating mean of stocks for each year:
- This helps us understand net stock price for each year for easier visualization.
In [9]:
#Finding mean for each of the open, high, low and closing values of the stocks for each year
apple_df_year = apple_df.groupby(apple_df.index.year).mean().round(2)
#Displaying the most recent years that is the ending rows of the dataframe
apple_df_year.tail()
Out[9]:
- Above shown is the mean of all the stock prices(open, high, low, close) of Apple grouped by Year.
- We can see that the Average Stock Prices go up every year.
- We can also notice that there has been quite a huge increase in stock prices per share from mean value of 120 dollars per share in 2015 to 196 dollars per share in 2019.
In [10]:
#Using the pandas dataframe and the decribe method we can see the descriptive statistics
print("Apple Stock Descriptive Stats:")
apple_df.describe().round(2)
Out[10]:
- The above tables gives us the descriptive statistics of Apple stock market showing different statistics like mean, standard deviation, minimum and maximum values.
- As we can notice that the mean prices of Apple stock has been around 123 dollars per share and the maximum value of the stock price has been 249 dollars per share.
Task 3.2: Data Analysis: Data Visualization¶
Characterising and Visualizing time series:
- Using matplotlib, I visualized the time series in order to gain characterisitic insights on daily, monthly, quarterly and annual frequencies.
- Discussing how each time series is changing over time
In [11]:
#Plotting visualization of time series to see change rate of stock price annually
apple_df["2012":"2019"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2)
#Naming the plot
plt.title('Apple Stock Price (Annual Frequency)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the annual trend of stock prices for the company Apple from the year 2012-2019.
- We notice that the overall trend of the closing stock prices is increasing
- However, it is interesting to notice the dip in the low stock prices in the middle of 2015 and also towards the end of each year and a strict rise from 2017 onwards.
- It is also interesting to notice the fall in stock prices nearing the end of 2018 followed by a sudden increase following the year 2019.
- Let us visualize the 2018-19 in terms of a monthly frequency to gain more insights.
In [12]:
#Plotting visualizations of time series to see change rate of stock price monthly
apple_df["2018-01":"2019-04"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2)
#Naming the plot
plt.title('Apple Stock Price (Monthly Frequency)')
#Labelling the axis
plt.xlabel("Month",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the monthly trend of stock prices for the company Apple from March 2018 to March 2019
- We notice that the overall trend of the closing stock prices is seasonal affected by the chosen time period. We can see rise and falls in particular months.
- We can see that from July 2018 there has been a steep rise in the stock prices and following the end of 2018, there has been a decreasing trend.
- Following January 2019, there has been a strict increase
- Let us visualize the quarter from December 2018 to February 2019 in terms of a quarterly frequency to gain more insights.
In [13]:
#Plotting visualization of time series to see change rate of stock price quarterly
apple_df["2018-12":"2019-02"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 3)
#Naming the plot
plt.title('Apple Stock Price (Quarterly Frequency)')
#Labelling the axis
plt.xlabel("Quarter",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the quaterly trend of stock prices for the company Apple from December 2018 to February 2019
- We notice that the trend of the prices have been drecreasing for the first half of the quarter and increasing after January 2019.
- The interesting part is the sudden fall in the prices in the first few days of January 2019 and an increase following those dates.
- The trend following February 2019 has been more or less constant.
- Let us visualize the daily frequency from January 2019 to February 2019 to gain more insights.
In [14]:
#Plotting visualization of time series to see change rate of stock price daily
apple_df["2019-01-01":"2019-02-01"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 3)
#Naming the plot
plt.title('Apple Stock Price (Daily Frequency)')
#Labelling the axis
plt.xlabel("Day",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the daily trend of stock prices for the company Apple from January 2019 to February 2019
- We notice that overall that has been an increasing trend for the stock prices for each day of the month.
- We can even notice several breaking points such as peaks and lows in a triangular increase fashion and a peak on the 29th of January.
Task 3.3: Quantifying and Discussing the time series¶
Discussing the changes in time series:
- Using matplotlib, I used different graphs like bar graphs and rolling mean line graphs, I discussed the plots in details to get more insights.
In [15]:
#Plotting bar graph visualization of time series to see change rate of stock price annually
apple_df_year.plot(kind = 'bar', figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, stacked = False, zorder=3)
#Naming the plot
plt.title('Apple Stock Price (Annual bar chart)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows that overall there has been a huge increase in the stock prices of Apple from 80 dollars per share in the year 2012 to 195 dollars per share in the year 2019.
In [16]:
#Calculating 30 year rolling mean of APPLE stock prices
rm = apple_df["Close"].rolling(30).mean()
p = rm.plot(figsize=(16, 5), fontsize=13, grid = True, color= 'red', linewidth = 3)
#Naming the plot
plt.title('Apple Stock Price (Rolling mean of 30 years)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13)
Out[16]:
- The above plot shows the rolling mean line graph for the company Apple.
- Rolling means (or moving averages) are generally used to smooth out short-term fluctuations in time series data and highlight long-term trends.
- We are now able to see a visualization which is without any noise.
- Clearly we can notice that there was a huge fall in stock prices of Apple nearing 2019 which I would be discussing in detail in my outlying observations.
Task 3.4: Outlying observations & Final conclusions¶
Observations:
- Through a candlestick plot made using plotly library, I discussed all the outlying observations throughout the changes in the time series.
In [17]:
#Using plotly library, plotting a candlestick plot to discuss the increasing and decreasing points of the stock prices
fig = go.Figure(data=[go.Candlestick(x=apple_df.index,
open=apple_df['Open'],
high=apple_df['High'],
low=apple_df['Low'],
close=apple_df['Close'], increasing_line_color= 'cyan', decreasing_line_color= 'grey')])
fig.update_layout(
#naming the plot
title='Apple Stock Price',
#labelling the axis
yaxis_title='Price($)',
xaxis_title='Year',
shapes = [dict(
#labelling the increasing period line
x0='2016-12-09', x1='2016-12-09', y0=0, y1=1, xref='x', yref='paper',
line_width=2)],
annotations=[dict(
x='2016-12-09', y=0.05, xref='x', yref='paper',
showarrow=False, xanchor='left', text='Increase Period Begins')]
)
fig.show()
12% Fall in Apple stocks in December 2018
- We notice that there was a strict increase in the Apple stock price following 2017, however a huge dip was noticed in December 2018.
- The stock ended trading at 142.12 dollars per share in the year 2018 which has been quite low since 2017.
- According to CNBC, there had been a lot of concern about how China-U.S. trade relations might impact iPhone sales. On top of that, in early December, a Chinese court ordered Apple to halt sales of older iPhone models. China accounted for about a fifth of Apple's total revenue in fiscal 2018 (which ended in September).
- At least a few analysts have cut their price targets on Apple over the last month, which can affect the stock's performance in the short term
- Now the iPhone maker has also been infected by the market fears, of which there are plenty, including overly optimistic valuations, rising interest rates, currency fluctuation and US-China trade tensions.
A Huge Increase in 2019
- However, an analyst predicted that Apple's stock could increase upto 13% in 2019 which has proven to be right.
- Apple shares have made a strong comeback in 2019. After falling more than 8% in 2018, the stock has risen close to 40% YTD (year-to-date).
- The iPhone11 continues to account for close to 50% of the company’s total sales. Apple will be banking on growing iPhone demand from China. Nearly 70 million customers in China are due for an upgrade. -Apple hasn’t increased the price of its new iPhones. In fact, the iPhone 11 is priced at 699 dollars, while the Pro and Max are available starting at 999 dollars and 1090 dollars, respectively. The iPhone 11 will be a key revenue driver for the next 12–18 months.
- Finally, there's Apple's fast-growing wearables business. Its wearables products -- Apple Watch, AirPods, and Beats earphones -- have taken off.
- It is predicted that Apple stock could close on 300 dollars per share in 2019 and have estimated to be in good shape even in the coming year of 2020.
Alphabet Inc.
- Stock analysis for the company Alphabet Inc. [Parent company of Google]
Task 1: Data Collection¶
- Using python, I downloaded the HTML pages for my chosen stocks
In [18]:
#calling the method get_raw_data to download data for the company Alphabet Inc
get_raw_data("goog")
Task 2.1: Data Preparation: Parsing and extracting data¶
Parsing data:
- Using pandas libray, I parsed and extracted the data for the company Alphabet Inc. and represented the whole dataset as a dataframe
In [19]:
#Creating the Alphabet dataframe and concatenting all the columns into one dataframe
google_df = pd.read_html("http://mlg.ucd.ie/modules/COMP30760/stocks/goog.html")
google_df = pd.concat(google_df)
#Displaying the dataframe
google_df
Out[19]:
Representing data as time series:
- To represent data as time series, first I would combine the year, month and day column of the data frame
- Use the date column as the index of the data frame
- Display the dataframe
In [20]:
#representing data as time series
google_df['Date'] = pd.to_datetime(google_df[['Year', 'Month', 'Day']])
#setting the data column as index
google_df.set_index('Date', inplace =True)
google_df.head()
Out[20]:
Exporting data to an appropriate format:
- Exporting the pre procesed data into csv file
In [21]:
#Exporting the data to csv file
google_df.to_csv("Alphabet Inc. Stock.csv", index=True, encoding='utf8')
In [22]:
#Dropping the the unnecessary columns
google_df = google_df.drop(["Stock", "Year", "Month", "Day"], axis=1)
#Diplaying the data frame
google_df.head()
Out[22]:
Handling missing values:
- Using interpolate method in the pandas library, I filled in the missing values in the dataframe
In [23]:
#Using df.interpolate, we can fill in the missing values in the dataframe
google_df = google_df.interpolate()
#Check if there are still any missing values left
google_df.isnull().sum()
Out[23]:
In [24]:
#Rounding off the stock figures to make sure there is consistency in the data
google_df = google_df.round(2)
google_df.head()
Out[24]:
Pre processing completed:
- The data contains no null values i.e there are no values in the dataset such as "Not Available", "NaN or "NA"
- All the values are not strings either
- This indicates that there all the values are present in the dataset.
Task 3.1: Data Analysis: Summarising the cleaned dataset¶
Descriptive Statistics:
- Displaying the descriptive statistics of the dataset
- Using the cleaned dataset for further analysis and visualization
- Grouping the data by year and taking mean of the values and displaying descriptive statistics
Calculating mean of stocks for each year:
- This helps us understand net stock price for each year for easier visualization.
In [25]:
#Finding mean for each of the open, high, low and closing values of the stocks for each year
google_df_year = google_df.groupby(google_df.index.year).mean().round(2)
#Displaying the most recent years that is the ending rows of the dataframe
google_df_year.tail()
Out[25]:
- Above shown is the mean of all the stock prices(open, high, low, close) of Alphabet Inc. grouped by Year.
- We can see that the Average Stock Prices go up every year.
- Also notice that there has been quite a huge increase in stock prices per share from mean value of 601 dollars per share in 2015 to 1162 dollars per share in 2019.
In [26]:
#Using the pandas dataframe and the decribe method we can see the descriptive statistics
print("Alphabet Inc. Stock Descriptive Stats:")
google_df.describe().round(2)
Out[26]:
- The above tables gives us the descriptive statistics of Alphabet Inc. stock market showing different statistics like mean, standard deviation, minimum and maximum values.
- As we can notice that the mean prices of Alphabet stock has been around 724 dollars per share and the maximum value of the stock price has been 1290 dollars per share.
Task 3.2: Data Analysis: Visualizing Data¶
Characterising and Visualizing time series:
- Using matplotlib, I visualized the time series in order to gain characterisitic insights on daily, monthly, quarterly and annual frequencies.
In [27]:
#Plotting visualization of time series to see change rate of stock price annually
google_df["2012":"2019"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['#3FC5F0', '#4DD599','#8F71FF', '#ff5d9e'])
#Naming the plot
plt.title('Alphabet Inc. Stock Price (Annual Frequency)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the annual trend of stock prices for the company Alphabet Inc. from the year 2012-2019.
- We notice that the overall trend of the closing stock prices is increasing
- The trend has been strickly rising from the year 2015 onwards and it is also quite instresting to notice the pattern from 2018 to 2019 onwards, it has been a inverse head and shoulders pattern, which is showing the reversal of a downward trend, that is increase in stock prices following the yer 2019.
- Let us visualize the 2018-19 in terms of a monthly frequency to gain more insights.
In [28]:
#Plotting visualizations of time series to see change rate of stock price monthly
google_df["2018-01":"2019-04"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['#3FC5F0', '#4DD599','#8F71FF', '#ff5d9e'])
#Naming the plot
plt.title('Alphabet Inc. Stock Price (Monthly Frequency)')
#Labelling the axis
plt.xlabel("Month",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the monthly trend of stock prices for the company Alphabet Inc. from March 2018 to March 2019
- We notice that the overall trend of the closing stock prices is seasonal affected by the chosen time period. We can see rise and falls in particular months.
- We can see that from July 2018 there has been a steep rise in the stock prices and following the end of 2018, there has been a decreasing trend.
- Following January 2019, there has been a strict increase
- This is very similar to the Apple stock prices trend.
- Let us visualize the quarter from December 2018 to February 2019 in terms of a quarterly frequency to gain more insights.
In [29]:
#Plotting visualization of time series to see change rate of stock price quarterly
google_df["2018-12":"2019-02"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 3, color = ['#3FC5F0', '#4DD599','#8F71FF', '#ff5d9e'])
#Naming the plot
plt.title('Alphabet Inc. Stock Price (Quarterly Frequency)')
#Labelling the axis
plt.xlabel("Quarter",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the quaterly trend of stock prices for the company Alphabet Inc. from December 2018 to February 2019
- We notice that the trend of the prices have been drecreasing nearing the end of 2018 and increasing after January 2019.
- The interesting part is the sudden fall in the prices in the first few days of January 2019 and an increase following those dates. This is very similar to the Apple trend as we saw previously
- The trend following February 2019 has been more or less constant.
- Let us visualize the daily frequency from January 2019 to February 2019 to gain more insights.
In [30]:
#Plotting visualization of time series to see change rate of stock price daily
google_df["2019-01-01":"2019-02-01"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 3, color = ['#3FC5F0', '#4DD599','#8F71FF', '#ff5d9e'])
#Naming the plot
plt.title('Alphabet Inc. Stock Price (Daily Frequency)')
#Labelling the axis
plt.xlabel("Day",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the daily trend of stock prices for the company Alphabet Inc. from January 2019 to February 2019
- We notice that overall that has been an increasing trend for the stock prices for each day of the month.
Task 3.3: Quantifying and Discussing the time series¶
Discussing the changes in time series:
- Using matplotlib, I used different graphs like bar graphs and rolling mean line graphs, I discussed the plots in details to get more insights.
In [31]:
#Plotting bar graph visualization of time series to see change rate of stock price annually
google_df_year.plot(kind = 'bar', figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, stacked = False, zorder=3,color = ['#3FC5F0', '#4DD599','#8F71FF', '#ff5d9e'] )
#Naming the plot
plt.title('Alphabet Inc. Stock Price (Annual bar chart)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows that overall there has been a huge increase in the stock prices of Alphabet Inc. from around 350 dollars per share in the year 2012 to a massive 1190 dollars per share in the year 2019.
In [32]:
#Calculating 30 year rolling mean of ALPHABET INC. stock prices
rm = google_df["Close"].rolling(30).mean()
p = rm.plot(figsize=(16, 5), fontsize=13, grid = True, color= '#ff5d9e', linewidth = 3)
#Naming the plot
plt.title('Alphabet Inc. Stock Price (Rolling mean of 30 years)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13)
Out[32]:
- The above plot shows the rolling mean line graph for the company Alphabet Inc.
- Rolling means (or moving averages) are generally used to smooth out short-term fluctuations in time series data and highlight long-term trends.
- We are now able to see a visualization which is without any noise.
- Clearly we can notice that there was a slight fall in the stocks nearing 2019 which I would be discussing in detail in my outlying observations.
Task 3.4: Outlying observations & Final conclusions¶
Observations:
- Through a candlestick plot made using plotly library, I discussed all the outlying observations throughout the changes in the time series.
In [33]:
#Using plotly library, plotting a candlestick graph to dicsuss the increasing and decreasing points of the stock prices
fig = go.Figure(data=[go.Candlestick(x=google_df.index,
open=google_df['Open'],
high=google_df['High'],
low=google_df['Low'],
close=google_df['Close'], increasing_line_color= '#4DD599', decreasing_line_color= '#ff5d9e')])
fig.update_layout(
#Naming the plot
title='Alphabet Inc. Stock Price',
yaxis_title='Price($)',
xaxis_title='Year',
#Labelling the increasing and decreasing patterns
shapes = [dict(
x0='2017-01-09', x1='2017-01-09', y0=0, y1=1, xref='x', yref='paper',
line_width=2)],
annotations=[dict(
x='2017-01-09', y=0.05, xref='x', yref='paper',
showarrow=False, xanchor='left', text='Increase Period Begins')]
)
fig.show()
Reason for fall in 2018
- According to maketwatch, The stock started its last bear market on Nov. 19, 2018, bottomed at a more than 14-month low of 984.67 dollars on Dec. 24, 2018, then ended the bear market on March 12, 2019 after rallying more than 20% off the bottom.
- The world’s dominant provider of internet search, advertising, and video services, Alphabet has increased spending in recent years on areas including cloud computing and consumer electronics that it views as essential to maintaining its industry leadership. The disappointing earnings come as Google faces major challenges internally and from outside regulators.
2019: Rise
- However, after the fall in stock prices in 2018, we can notice the stocks rising up in 2019 again.
- Google, Alphabet's subsidiary is to benefit from digital ad revenue at an exponential rate, therefore analysts recomment in buying Alphabet's stock.
- Google continues to grow its “other revenues” segment, which includes its cloud business and hardware sales. The division accounted for 6.49 billion dollars during the quarter, narrowly beating Wall Street estimates of 6.43 billion dollars.
- That marks a 31 percent increase year over year. The company declined to break out Cloud revenues for the quarter, but said it remains “one of the fastest growing businesses across Alphabet.”
Disney
- Stock analysis for the company Intel
Task 1: Data Collection¶
- Using python, I downloaded the HTML pages for my chosen stocks
In [34]:
#calling the method get_raw_data to download data for the company DISNEY
get_raw_data("dis")
Task 2.1: Data Preparation: Parsing and extracting data¶
Parsing data:
- Using pandas libray, I parsed and extracted the data for the company DISNEY and represented the whole dataset as a dataframe
In [35]:
#Creating the Disney dataframe and concatenting all the columns into one dataframe
disney_df = pd.read_html("http://mlg.ucd.ie/modules/COMP30760/stocks/dis.html")
disney_df = pd.concat(disney_df)
#Displaying the dataframe
disney_df
Out[35]:
Representing data as time series:
- To represent data as time series, first I would combine the year, month and day column of the data frame
- Use the date column as the index of the data frame
- Display the dataframe
In [36]:
#representing data as time series
disney_df['Date'] = pd.to_datetime(disney_df[['Year', 'Month', 'Day']])
#setting the data column as index
disney_df.set_index('Date', inplace =True)
disney_df.head()
Out[36]:
Exporting data to an appropriate format:
- Exporting the data into csv file
In [37]:
#Exporting the dataframe to csv file
disney_df.to_csv("Disney Stock.csv", index=True, encoding='utf8')
In [38]:
#Dropping the the unnecessary columns
disney_df = disney_df.drop(["Stock", "Year", "Month", "Day"], axis=1)
#Diplaying the data frame
disney_df.head()
Out[38]:
Handling missing values:
- Using interpolate method in the pandas library, I filled in the missing values in the dataframe
In [39]:
#Using df.interpolate, we can fill in the missing values in the dataframe
disney_df = disney_df.interpolate()
#Check if there are still any missing values left
disney_df.isnull().sum()
Out[39]:
In [40]:
#Rounding off the stock figures to make sure there is consistency in the data
disney_df = disney_df.round(2)
disney_df.head()
Out[40]:
Pre processing completed:
- The data contains no null values i.e there are no values in the dataset such as "Not Available", "NaN or "NA"
- All the values are not strings either
- This indicates that there all the values are present in the dataset.
Task 3.1: Data Analysis: Summarising the cleaned dataset¶
Descriptive Statistics:
- Displaying the descriptive statistics of the dataset
- Using the cleaned dataset for further analysis and visualization
- Grouping the data by year and taking mean of the values and displaying descriptive statistics
Calculating mean of stocks for each year:
- This helps us understand net stock price for each year for easier visualization.
In [41]:
#Finding mean for each of the open, high, low and closing values of the stocks for each year
disney_df_year = disney_df.groupby(disney_df.index.year).mean().round(2)
#Displaying the most recent years that is the ending rows of the dataframe
disney_df_year.tail()
Out[41]:
- Above shown is the mean of all the stock prices(open, high, low, close) of Disney grouped by Year.
- We can see that the Average Stock Prices go up every year but there was a slight decrese in the year 2016 after which there was a very slight increase as compared to Alphabet Inc. and Apple.
- Overall the mean of the closing values of stock prices in 2019 have been higher than the previous years.
In [42]:
#Using the pandas dataframe and the decribe method we can see the descriptive statistics
print("Disney Stock Descriptive Stats:")
disney_df.describe().round(2)
Out[42]:
- The above tables gives us the descriptive statistics of Disney stock market showing different statistics like mean, standard deviation, minimum and maximum values.
- As we can notice that the mean prices of Disney stock has been around 92 dollars per share and the maximum value of the stock price has been 146 dollars per share.
Task 3.2: Data Analysis: Visualizing Data¶
Characterising and Visualizing time series:
- Using matplotlib, I visualized the time series in order to gain characterisitic insights on daily, monthly, quarterly and annual frequencies.
In [43]:
#Plotting visualization of time series to see change rate of stock price annually
disney_df["2012":"2019"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['Green', 'Yellow','Red', 'Blue'])
#Naming the plot
plt.title('Disney Stock Price (Annual Frequency)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the annual trend of stock prices for the company Disney from the year 2012-2019.
- We notice that the overall trend of the closing stock prices is increasing but the trend contains many cup and handles and several patterns.
- The trend has been a steady increase until 2016 followed by a cup and handle pattern nearing 2016 and a inverse head and shoulders pattern nearing 2017.
- We can also see that just like Apple and Alphabet, there is a similar fall in disney stock prices in 2019.
- Let us visualize the 2018-19 in terms of a monthly frequency to gain more insights.
In [44]:
#Plotting visualizations of time series to see change rate of stock price monthly
disney_df["2018-01":"2019-04"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['Green', 'Yellow','Red', 'Blue'])
#Naming the plot
plt.title('Disney Stock Price (Monthly Frequency)')
#Labelling the axis
plt.xlabel("Month",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the monthly trend of stock prices for the company Disney from March 2018 to March 2019
- We notice that the overall trend of the closing stock prices is seasonal affected by the chosen time period. We can see rise and falls in particular months.
- We can see that the trend repeats itself in cup and handle fashion.
- However, it is interesting to notice the strict increase from March 2019 which resulted in Disney stocks worth 140 dollars per share.
- Let us visualize the quarter from December 2018 to February 2019 in terms of a quarterly frequency to gain more insights.
In [45]:
#Plotting visualizations of time series to see change rate of stock price quarterly
disney_df["2018-12":"2019-02"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['Green', 'Yellow','Red', 'Blue'])
#Naming the plot
plt.title('Disney Stock Price (Quarterly Frequency)')
#Labelling the axis
plt.xlabel("Quarter",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the quaterly trend of stock prices for the company Disney from December 2018 to February 2019
- We notice that the trend of the prices have been drecreasing nearing the end of 2018 and increasing after January 2019.
- The interesting part is the sudden fall in the prices in the first few days of January 2019 and an increase following those dates. This is very similar to the Apple and Alphabet trend as we saw previously
- The trend following February 2019 has been more or less constant.
- Let us visualize the daily frequency from January 2019 to February 2019 to gain more insights.
In [46]:
#Plotting visualizations of time series to see change rate of stock price quarterly
disney_df["2019-01-01":"2019-02-01"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['Green', 'Yellow','Red', 'Blue'])
#Naming the plot
plt.title('Disney Stock Price (Daily Frequency)')
#Labelling the axis
plt.xlabel("Day",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the daily trend of stock prices for the company Disney from January 2019 to February 2019
- We notice that overall that has been an increasing trend for the stock prices followed by a slight decrease in trend nearing february and a spike on the 1st of February 2019.
Task 3.3: Quantifying and Discussing the time series¶
Discussing the changes in time series:
- Using matplotlib, I used different graphs like bar graphs and rolling mean line graphs, I discussed the plots in details to get more insights.
In [47]:
#Plotting bar graph visualization of time series to see change rate of stock price annually
disney_df_year.plot(kind = 'bar', figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, stacked = False, zorder=3, color = ['Green', 'Yellow','Red', 'Blue'])
#Naming the plot
plt.title('Disney Stock Price (Annual bar chart)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows that overall there has been a huge increase in the stock prices of Disney from around 50 dollars per share in the year 2012 to a massive 130 dollars per share in the year 2019.
In [48]:
#Calculating 30 year rolling mean of DISNEY stock prices
rm = disney_df["Close"].rolling(30).mean()
p = rm.plot(figsize=(16, 5), fontsize=13, grid = True, color= 'Blue', linewidth = 3)
#Naming the plot
plt.title('Disney Stock Price (Rolling mean of 30 years)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13)
Out[48]:
- The above plot shows the rolling mean line graph for the company Disney.
- We can notice the cup and handle pattern and that there was a slight fall in the stocks nearing 2019 which I would be discussing in detail in my outlying observations.
Task 3.4: Outlying observations & Final conclusions¶
Observations:
- Through a candlestick plot made using plotly library, I discussed all the outlying observations throughout the changes in the time series.
In [49]:
#Using plotly library, plotting a candlestick plot to discuss the increasing and decreasing points of the stock prices
fig = go.Figure(data=[go.Candlestick(x=disney_df.index,
open=disney_df['Open'],
high=disney_df['High'],
low=disney_df['Low'],
close=disney_df['Close'], increasing_line_color= 'blue', decreasing_line_color= 'red')])
fig.update_layout(
#naming the plot
title='Disney Stock Price',
#labelling the axis
yaxis_title='Price($)',
xaxis_title='Year',
shapes = [dict(
#labelling the increasing period line
x0='2016-11-09', x1='2016-11-09', y0=0, y1=1, xref='x', yref='paper',
line_width=2)],
annotations=[dict(
x='2016-11-09', y=0.05, xref='x', yref='paper',
showarrow=False, xanchor='left', text='Increase Period Begins')]
)
fig.show()
2018 December: Disney stocks fall to about 3.71%
- According to equities.com fter a few months of stock market turbulence, Disney is down 11% from the multiyear highs it reached over the fall of 2018. It's not that Disney itself did anything to upset investors; on the contrary, the company is nearing the closure of its takeover of Twenty-First Century Fox.
- Among the S&P 500’s biggest fallers on Monday December 24 was The Walt Disney Company. The stock experienced a 3.71% decline to 100.35 dollars with 7.13 million shares changing hands. The Walt Disney Company started at an opening price of 103.23 and hit a high of 103.90 and a low of 100.35. Ultimately, the stock took a hit and finished the day at 3.87 dollars per share.
The 2019 rise
- In 2019, the Disney stocks noticed a spike.
- The increase in Walt Disney stock price was primarily driven by an increase in revenue and margins, along with a sharp drop of 202 million in shares outstanding which has pushed up the earnings per share. However, this momentum was partially offset by a reduction of 10.9x in the P/E multiple during this period. Despite recent hiccups with respect to margins, as per Walt Disney Valuation by Trefis, we have a price estimate of 152 per share for DIS’ stock, higher than its current market price.
- Key revenue drivers include parks, resorts and studio entertainment. Domestic revenue growth is driven by higher average ticket prices for theme park admissions and for cruise line sailings, increased food, beverage, and merchandise spending, and higher average daily hotel room rates, along with higher volume due to higher attendance and passenger cruise ship days. Studio Entertainment revenue is expected to grow at a faster rate in the near term, driven by higher theatrical distribution and TV/SVOD revenues, partially offset by continued pressure in home entertainment. Also, the acquisition of FOX and the kickstart of Disney plus played a vital role in increasing the revnue of company to a huge extent.
Microsoft
- Stock analysis for the company Microsoft
Task 1: Data Collection¶
- Using python, I downloaded the HTML pages for my chosen stocks
In [50]:
#calling the method get_raw_data to download data for the company MICROSOFT
get_raw_data("msft")
Task 2.1: Data Preparation: Parsing and extracting data¶
Parsing data:
- Using pandas libray, I parsed and extracted the data for the company MICROSOFT and represented the whole dataset as a dataframe
In [51]:
#Creating the Microsoft dataframe and concatenting all the columns into one dataframe
microsoft_df = pd.read_html("http://mlg.ucd.ie/modules/COMP30760/stocks/msft.html")
microsoft_df = pd.concat(microsoft_df)
#Displaying the dataframe
microsoft_df
Out[51]:
Representing data as time series:
- To represent data as time series, first I would combine the year, month and day column of the data frame
- Use the date column as the index of the data frame
- Display the dataframe
In [52]:
#representing data as time series
microsoft_df['Date'] = pd.to_datetime(microsoft_df[['Year', 'Month', 'Day']])
#setting the data column as index
microsoft_df.set_index('Date', inplace =True)
microsoft_df.head()
Out[52]:
Exporting data to an appropriate format:
- Exporting the data into csv file
In [53]:
#Exporting the raw data to csv file with Date as index
microsoft_df.to_csv("Microsoft Stock.csv", index=True, encoding='utf8')
In [54]:
#Dropping the the unnecessary columns
microsoft_df = microsoft_df.drop(["Stock", "Year", "Month", "Day"], axis=1)
#Diplaying the data frame
microsoft_df.head()
Out[54]:
Handling missing values:
- Using interpolate method in the pandas library, I filled in the missing values in the dataframe
In [55]:
#Using df.interpolate, we can fill in the missing values in the dataframe
microsoft_df = microsoft_df.interpolate()
#Check if there are still any missing values left
microsoft_df.isnull().sum()
Out[55]:
In [56]:
#Rounding off the stock figures to make sure there is consistency in the data
microsoft_df = microsoft_df.round(2)
microsoft_df.head()
Out[56]:
Pre processing completed:
- The data contains no null values i.e there are no values in the dataset such as "Not Available", "NaN or "NA"
- All the values are not strings either
- This indicates that there all the values are present in the dataset.
Task 3.1: Data Analysis: Summarising the cleaned dataset¶
Descriptive Statistics:
- Displaying the descriptive statistics of the dataset
- Using the cleaned dataset for further analysis and visualization
- Grouping the data by year and taking mean of the values and displaying descriptive statistics
Calculating mean of stocks for each year:
- This helps us understand net stock price for each year for easier visualization.
In [57]:
#Finding mean for each of the open, high, low and closing values of the stocks for each year
microsoft_df_year = microsoft_df.groupby(microsoft_df.index.year).mean().round(2)
#Displaying the most recent years that is the ending rows of the dataframe
microsoft_df_year.tail()
Out[57]:
- Above shown is the mean of all the stock prices(open, high, low, close) of Microsoft grouped by Year.
- We can see that the Average Stock Prices go up every year.
- We can also notice that there has been quite a huge increase in stock prices per share from mean value of 46 dollars per share in 2015 to 126 dollars per share in 2019.
In [58]:
#Using the pandas dataframe and the decribe method we can see the descriptive statistics
print("Microsoft Stock Descriptive Stats:")
microsoft_df.describe().round(2)
Out[58]:
- The above tables gives us the descriptive statistics of Microsoft stock market showing different statistics like mean, standard deviation, minimum and maximum values.
- As we can notice that the mean prices of Microsoft stock has been around 61 dollars per share and the maximum value of the stock price has been 144 dollars per share.
Task 3.2: Data Analysis: Visualizing Data¶
Characterising and Visualizing time series:
- Using matplotlib, I visualized the time series in order to gain characterisitic insights on daily, monthly, quarterly and annual frequencies.
In [59]:
#Plotting visualization of time series to see change rate of stock price annually
microsoft_df["2012":"2019"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['#ffba01', '#01a6f0','#f34f1c', '#7fbc00'])
#Naming the plot
plt.title('Microsoft Stock Price (Annual Frequency)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the annual trend of stock prices for the company Microsoft from the year 2012-2019.
- We notice that the overall trend of the closing stock prices is increasing and it is interesting to observe the two cup pattern in the year 2015-16.
- We can also see that just like all the other companies above the stock prices fall in nearing 2019 and then shoot up.
- Let us visualize the 2018-19 in terms of a monthly frequency to gain more insights.
In [60]:
#Plotting visualization of time series to see change rate of stock price monthly
microsoft_df["2018-01":"2019-04"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['#ffba01', '#01a6f0','#f34f1c', '#7fbc00'])
#Naming the plot
plt.title('Microsoft Stock Price (Monthly Frequency)')
#Labelling the axis
plt.xlabel("Month",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the monthly trend of stock prices for the company Microsoft from March 2018 to March 2019
- We notice that the overall trend of the closing stock prices is seasonal affected by the chosen time period. We can see rise and falls in particular months.
- However, it is interesting to notice the strict increase from January 2019 which resulted in Microsoft stocks worth 130 dollars per share.
- Let us visualize the quarter from December 2018 to February 2019 in terms of a quarterly frequency to gain more insights.
In [61]:
#Plotting visualizations of time series to see change rate of stock price quarterly
microsoft_df["2018-12":"2019-02"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2,color = ['#ffba01', '#01a6f0','#f34f1c', '#7fbc00'] )
#Naming the plot
plt.title('Microsoft Stock Price (Quarterly Frequency)')
#Labelling the axis
plt.xlabel("Quarter",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows the quaterly trend of stock prices for the company Microsoft from December 2018 to February 2019
- We notice that the trend of the prices have been drecreasing nearing the end of 2018 and increasing after January 2019.
The interesting part is the sudden fall in the prices in the first few days of January 2019 and an increase following those dates. This is very similar to the Apple and Alphabet trend as we saw previously
The plot below shows a daily trend for the Microsoft stocks from January 2019 to February 2019 and we notice an increasing trend follows by a slight decrease nearing 1st of February.
In [62]:
#Plotting visualization of time series to see change rate of stock price daily
microsoft_df["2019-01-01":"2019-02-01"].plot(figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, color = ['#ffba01', '#01a6f0','#f34f1c', '#7fbc00'])
#Naming the plot
plt.title('Microsoft Stock Price (Daily Frequency)')
#Labelling the axis
plt.xlabel("Day",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
Task 3.3: Quantifying and Discussing the time series¶
Discussing the changes in time series:
- Using matplotlib, I used different graphs like bar graphs and rolling mean line graphs, I discussed the plots in details to get more insights.
In [63]:
#Plotting bar graph visualization of time series to see change rate of stock price annually
microsoft_df_year.plot(kind = 'bar', figsize=(16, 5), fontsize=13, grid= True, linewidth = 2, stacked = False, zorder=3,color = ['#ffba01', '#01a6f0','#f34f1c', '#7fbc00'] )
#Naming the plot
plt.title('Microsoft Stock Price (Annual bar chart)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13);
- The plot above shows that overall there has been a huge increase in the stock prices of Microsoft from around 30 dollars per share in the year 2012 to a massive 125 dollars per share in the year 2019.
In [64]:
#Calculating 30 year rolling mean of MICROSOFT stock prices
rm = microsoft_df["Close"].rolling(30).mean()
p = rm.plot(figsize=(16, 5), fontsize=13, grid = True, color= '#7fbc00', linewidth = 3)
#Naming the plot
plt.title('Microsoft Stock Price (Rolling mean of 30 years)')
#Labelling the axis
plt.xlabel("Year",fontsize=13)
plt.ylabel("Price($)",fontsize=13)
Out[64]:
- The above plot shows the rolling mean line graph for the company Microsoft.
- We can notice the strict increse and that there was a slight fall in the stocks nearing 2019 which I would be discussing in detail in my outlying observations.
Task 3.4: Outlying observations & Final conclusions¶
Observations:
- Through a candlestick plot made using plotly library, I discussed all the outlying observations throughout the changes in the time series.
In [65]:
#Using plotly library, plotting a candlestick plot to discuss the increasing and decreasing points of the stock prices
fig = go.Figure(data=[go.Candlestick(x=microsoft_df.index,
open=microsoft_df['Open'],
high=microsoft_df['High'],
low=microsoft_df['Low'],
close=microsoft_df['Close'], increasing_line_color= '#f34f1c', decreasing_line_color= '#7fbc00')])
fig.update_layout(
#naming the plot
title='Microsoft Stock Price',
#labelling the axis
yaxis_title='Price($)',
xaxis_title='Year',
shapes = [dict(
#labelling the increasing period line
x0='2016-11-09', x1='2016-11-09', y0=0, y1=1, xref='x', yref='paper',
line_width=2)],
annotations=[dict(
x='2016-11-09', y=0.05, xref='x', yref='paper',
showarrow=False, xanchor='left', text='Increase Period Begins')]
)
fig.show()
Fall in stocks: December 2018
- According to Bloomberg The Microsoft’s stock fell 11 percent in the three months ended in December, hitting a low point late in the year, as concerns emerged that tech spending was slowing, particularly in areas like PCs.
Reason for spike in 2019 January
- Even with the market turmoil toward the end of the year, shares of Microsoft delivered an 18.7% return to shareholders last year, according to data provided by S&P Global Market Intelligence. The software giant continued to make advances in its transformation to a subscription and services-based business, which powered an impressive acceleration in revenue growth last year.
- These results come as Microsoft is seeing impressive growth in its Azure cloud business, which grew 76% year over year in the fiscal first quarter ending in September. Microsoft is now the second largest cloud provider in the world. -Meanwhile, other areas of the company are performing well, too. The Productivity and Business Processes division grew revenue 19% in the fiscal first quarter, driven by growth in subscribers to Office 365, as well as a robust increase of 33% in revenue from LinkedIn.
- The largest revenue contributor last quarter was the more personal computing segment, which grew 15% year over year, led by growth in Windows (up 12%), gaming (up 44%), search advertising (up 17%), and Surface devices (up 14%).
- Overall, Microsoft is looking strong. Total revenue growth of 19% year over year in the first quarter of fiscal 2019 represented an acceleration over the year-ago quarter's 12% growth rate. Given the momentum, it's clear why the stock climbed 18% last year.
Task 3.5: Comparing the different time series¶
Observations:
- Through a line plot made using plotly library, I compared the 4 time series and gave insights on my final conclusions.
In [66]:
#Using plotly library to display the comparison between all the 4 time series.
fig = go.Figure()
#Trend for the company Apple
fig.add_trace(go.Scatter(x=apple_df.index, y=apple_df['Close'], name="AAPL Close",
line_color='red'))
#Trend for the company Alphabet Inc.
fig.add_trace(go.Scatter(x=google_df.index, y=google_df['Close'], name="GOOG Close",
line_color='#ff5d9e'))
#Trend for the company Disney
fig.add_trace(go.Scatter(x=disney_df.index, y=disney_df['Close'], name="DIS Close",
line_color='Blue'))
#Trend for the company Microsoft
fig.add_trace(go.Scatter(x=microsoft_df.index, y=microsoft_df['Close'], name="MSFT Close",
line_color='#7fbc00'))
#Forming the plot layout
fig.update_layout(
#Naming the plot
title='Overall Stock Price of all 4 companies',
#Naming the axis
yaxis_title='Price($)',
xaxis_title='Year',
width = 1000,
height = 500)
#Displaying the plot
fig.show()
Final Insights:
- The above plot shows the 4 different time series and stock data from the companies Apple, Alphabet Inc., Disney, Microsoft
- One of the first things I noticed is the vast difference between Alphabet Inc. stock prices and all the other companies. Alphabet Inc. has risen from aound 350 dollars per share of stock to 1200 and above which is a huge increase. We can aso see that Apple has performed well as well throughout the past few years.
- One more similarity between all the time series has been the fall in stock prices nearing December 2018 and then the sudden rise in January 2019 followed by a reverse head and shoulders pattern on the plot and the mutual increasing period of stocks fetched from my candlestick plots has been around the year 2017.
- 2018 was not a good year for the stock market. Since the beginning of the year, the Dow Jones Industrial Average has lost about 10 percent of its value, as did the S&P 500. The Nasdaq dropped roughly 8 percent.
- The vast majority of losses have come since October, when the stock market, which was experiencing the longest bull run in history, took a turn for the worst. The stock market is on pace for its worst December since 1931, but it also setrecord single-day gains Wednesday, when the Dow jumped by more than 1,000 points.
- The stock market woes come despite signs that the general economy is still doing well — with record low unemployment, strong GDP growth and relatively low inflation.
President Donald Trump’s trade war with China, the slowdown in global economic growth and concern that the Federal Reserve was raising interest rates too quickly all contributed to a pessimistic reaction from the stock market.
Stocks had their best January gains in more than 30 years, and that should mean 2019 will be a pretty good year for the market.
- The market has sprung back from December’s low, with the S&P gaining 15 percent since Dec. 26. Stocks did well through most of January 2019, but by the end of the month, a correction started.
- We can finally conclude that stocks are affected by a variety of reasons like US Trade War, valuations of the companies, revenue made by the company and the new products launched by the companies at different points of time.
Further Analysis:
- We can make use machine learning algorithms and python libraries like stocker to predict the stock pattern for 2020 or even the following months.
- This can make us get a better understanding of what to expect but then again since stock is affected by a lot of external factors, we cannot be 100% right.